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PUBLIC-KEY CRYPTOSYSTEM BASED ON INVARIANTS OF 
DIAGONALIZABLE GROUPS 

FRANTISEK MARKO, ALEXANDR N. ZUBKOV, AND MARTIN JURAS 


Abstract. We develop a public key cryptosystem based on invariants of di- 
agonalizable groups and investigate properties of such cryptosystem first over 
finite fields, then over number fields and finally over finite rings. We consider 
the security of these cryptosystem and show that it is necessary to restrict the 
set of parameters of the system to prevent various attacks (including linear 
algebra attacks and attacks based on Euclidean algorithm). 


Introduction 

A new idea for a public-key cryptosystem based on the invariant theory was 
proposed by Grigoriev in m- His original idea was later developed in the paper 
m- The last paragraph of the paper [10] reads as follows: 

’’The current state of the art in cryptography does not allow one to prove the 
security of cryptosystems; this is usually a question of belief in the difficulty of a 
revelant problem and a matter of experience (that is why it is not quite unusual to 
have a paper on cryptography without theorems, for example, this paper). Quite the 
opposite, one can expect a ’’disappointing” breaking of a particular cryptosystem. 
This can happen for any of the afforementioned examples (without solving the graph 
isomorphism problem, see the discussion above). On the other hand, such breaking 
could lead to interesting algorithms in the theory of group representations. Thus 
one can treat the above examples (and the general construction as a whole) just as 
a suggestion to play with cryptosystems based on the invariant theory.” 

The purpose of our paper is to develop and design a public-key cryptosystem 
based on invariants of diagonalizable groups. We go beyond the philosophy of 
the preceeding quote and design a concrete public-key cryptosystem, present an 
algorithm for its implementation and show how to break systems based on invariants 
of some groups. 

At first we consider these cryptosystems over finite fields F, then we investigate 
cryptosystems over fields of characteristic zero (in particular, over number fields), 
and finally we work with cryptosystems over finite rings (in particular, residue 
classes of number fields modulo an integer). Each part is distinguished by distinctive 
properties. For example, cyclicity of the multiplicative group plays the most 
important role over finite fields, the theory of divisors and factorization properties 
are most important for the number fields, and both properties are important when 
we work over finite rings. One property that remains valid in all cases is that 
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if G not cyclic then it produces more complicated (and secure) cryptosystems as 
compared to the case when C is cyclic. 

Finding an invariant of the group G is trivial in the finite field case, and what 
is challenging is to find one separating vectors from the given set S. We show a 
simple example when n = 2 for which the security of the cryptosystem is equivalent 
to the discrete logarithm assumption. Over number fields, the main problem is to 
find an invariant of G and the problem of the separation of elements of S can be 
neglected. The cryptosystems over finite rings combine features of the previous two 
cases and further investigation of their properties will be necessary. 

Finally, our work on this cryptosystem leads to an investigation of interesting 
mathematical problems related to the security of the invariant-based cryptosystem. 
Theoretical results about related mathematical concepts like minimal degrees of 
invariants and invariants of supergroups will appear in a separate paper |15j . 

1. Invariants of finitely-generated linear groups 

In this paper, we will consider only finitely generated groups G acting faithfully 
on a finite-dimensional vector space V = F" over a field F of arbitrary character¬ 
istic. Therefore we can asume that G C GL(V). From the very beginning, assume 
that the representation p : G ^ GL[V) is hxed, and the group G is given by a 
finite set of generators. With respect to the standard basis of V, each element g 
of G is therefore represented by an invertible matrix of size n x n, and g acts on 
vectors in V by matrix multiplication. 

Let F[V] = F[xi,... ,Xn] be the algebra of polynomial functions on GL{V). 
Then G acts on F[V] via gf{v) = /(g“^u), where g € G, f G F[V] and v gV . An 
invariant / of G is a polynomial / G F\V] which has a property that its values are 
the same on orbits of the group G. In other words, for every vector v gV and for 
every element g G G, we have f{gv) = f(v). We note that different representations 
of G lead to different invariants in general but this is not going to be a problem for 
us since our representation of G is fixed. We will denote the algebra of invariants 
of G by F[Vf. 

2. Public key-cryptosystem based on invariants 

We start by recalling the original idea of the public-key cryptosystem based on 
invariants from the paper [lOj and recalling its modification presented in m 

2.1. Design of cryptosystems based on invariants of groups. To design a 
cryptosystem, Alice needs to choose a finitely generated subgroup G of GL{V) for 
some vector space P = F" and a set {gi,... ,gs} of generators of G. Alice also 
chooses an n X n matrix a. Alice needs to know an invariant / of this representation 
of G. Depending on this invariant /, Alice chooses a set M = {uq; • ■ • ,'Cr-i} of 
vectors from V such that the set S = aM = {avo,... ,avr-i} is separated by 
the invariant /. This means that f{avi) ^ f{avj) whenever i ^ j. The set M 
represents messages Alice can receive and elements of the set S are bijectively 
assigned to elements of M. The sets S' is a part of the public key. 

Alice also chooses a set of randomly generated elements gi, ■ ■ ■ ,gm of G (say, by 
multiplying some of the given generators of G), which generates a subgroup of G 
that will be denoted by Gs- 
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Alice announces as a public key the set S representing possible messages, and the 
group H = a~^Gsa, conjugated to Gs, by announcing its generators hi = a~^gia 
for i = 1,..., m. 

In the first paper [10] its author assumes that the group G, its representation 
in GL{V) and the invariant / are in the public key. We refer to this setup as 
variant one. However, the version in paper m assumes that G, its representation 
in GL(V) and the invariant / are secret. We refer to this setup as variant two. We 
will comment on both variants later. 

For the encryption, every time Bob wants to transmit a message m G M, he 
looks up the correponding element of Ui G S' and chooses a randomly generated 
element h of the group H(hy multiplying some of the generators of H given as a 
public key). Then he computes u = hvi and transmits the vector u £ V to Alice. 

To decript the message, Alice first computes au and then applies the invariant 
/. If u = hvi, then f{au) = f{ahvi) = f{aa~^gavi) = f{gavi) = f{avi). Since 
a was chosen so that f{avi) ^ f{avj) whenever i ^ j, Alice can determine from 
the value of f{au) whether the symbol Vi and the corresponding message that was 
encrypted by Bob. 

Let us discuss briefly the choices of n, F, G and S. 

It appears that choosing large n would be better for the security of the cryp¬ 
tosystem but it would increase the expansion in size from plaintext to ciphertext. 

The bigger and more complicated the structure of F, the better for the security of 
the cryptosystem. Analogously, the more complicated structure of G, in particular 
G not cyclic would be preferred. 

Finally, we should choose the set S as large as possible for two reasons. The 
first reason is that larger set S shrink the number of invariants of G that separate 
elements of S and thus increases the security of the cryptosystem. The second 
reason is that larger set S decreases the ratio of the expansion in size from plaintext 
to ciphertext for the encryption using this cryptosystem. 

2.2. Previously described attacks on the cryptosystem based on invari¬ 
ants of groups. Let us note that it is important that during the encryption process 
by Bob he uses all generators hi for scrambling the message. If some generators 
are not involved, then to decode his message Charlie would succeed if he finds an 
invariant of a subgroup of H, which is an easier task. 

The attacks described below are mentioned in m and m- We are providing 
their description for the convenience of the reader and for further clarification. Also, 
these attacks we previously described only for the case when \M\ = 2 and we adapt 
them to the case when \M\ = r. 

To break the encryption, it is enough for Charlie to find any invariant f' of the 
group H that separates elements of S'. If r > 2 then we can replace this by a weaker 
condition. Namely, it is enough to find /' such that f'{u) = f'{vi) for a unique 
Tj G S. 

Indeed if Charlie computes f'{u) = f'ihvi) = f'{vi) and then compares f'{u) 
with f'{vi). If there is a unique vector Vi such that f'{u) = f{vi), then the message 
corresponding to Vi was sent by Bob. 

2.2.1. Variant one. Consider variant one of the cryptosystem - that is, the group 
G, its representation in GL(V) and an invariant / are known. We can assume 
that / is a homogeneous polynomial of degree d. In this case, it is known that 
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there is a homogeneous invariant /' of H of degree d that is of the form f'{v) = 
f{bv) for some matrix b G GL(V). Then /' is an invariant of H if and only if 
f{bhiv) = f'{hiv) = f'{v) = f{bv) for each generator hi, where i = 1,... ,m oi H. 
Comparing coefficients at monomials we obtain linear equations 

in variables (entries of b). Any solution of this system produces an invariant of 
H. 

Another possible way to attack the system is to find a matrix b G GL(V) such 
that bHb~^ C G. This technique is related to the conjugacy problem for matrix 
groups and the graph isomorphism problem. 

2.2.2. Variant two. In variant two of the cryptosystem, the group G, its represen¬ 
tation in GL{V) and the invariant / are secret. However, Charlie can attempt 
to hnd an invariant f directly by choosing a possible degree d and solving linear 
systems derived from the equations f'{hiv) = f'(v) for each generator hi, where 
i = 1.... ,m. This produces a linear system consisting of equations in 

the unknowns that are the coefficients at monomials in /'. 

Another approach is to find a matrix h G H such that hu = Vi for some i 
(attempting to recover the encryption done by Bob). This problem is related to 
the vector transporter problem and the graph isomorphism problem - see [lOj . Let 
us note that it was announced reently in [5] that the graph isomorphism problem 
can be solved in quasipolynomial time. 

3. Cryptosystems over finite fields F 

In this section we will discuss cryptosystems based in invariants of groups over 
hnite fields F. We will present concrete examples and show how the security of 
those cryptosystems is guaranteed if we assume computational hardness of the 
dicrete logarithm problem. 

3.1. n = 1. For simplicity, in the case n = I, we will assume that the cardinality 
of the set S' is 2. 

The case n = I is singular and it implies G G F. If there is a nonconstant 
invariant / = p{x) of G that attains the constant value c when evaluated on each 
element of G, then G is a subset of the set of roots of the polynomial p{x) — c. In 
particular, G is finite. 

Let F be a finite field GF{q) of characteristic p > 0 and cardinality q = p^. 
The set of non-zero elements F^ oi F with respect to the multiplication in F form 
a cyclic group generated by a primitive element a. Also, F is isomorphic to the 
splitting field of the polynomial x’^ — x = 0 over the prime field GF(jp). In particular, 
x"* — X is an invariant of F and x'^~^ is an invariant of attaining the value 1. 

Since F^ is cyclic, every subgroup Gd of F^ is also cyclic, generated by , 
given as a set of roots of x'^ — I = 0 for some d\{q — 1). The set of invariants of G is 
generated by x"^. li d ^ q — 1, we can choose vq and vi from F^ such that Vg vf. 
In particular, any choice xq G Gd and vi ^ Gd will do. Then the invariant f = x‘^ 
of G separates vq and vi. 

For setting up the corresponding cryptosystem over a finite field F we need to 
select an element g G F (that generates G) and find an exponent d' < q — 1 such 
that g‘^ =1. Then we can select for xq any element of G (that is a power of g). 
We also need to find an element vi G F that does not belong to G. Since we do 


PUBLIC-KEY CRYPTOSYSTEM BASED ON INVARIANTS 


5 


not know the order d oi g, the simplest way to guarantee this condition is to make 
sure that vf ^1. Then x‘^ is an invariant of G separating vg and vi. 

To break such a system, we need, for a given group G and vectors Vg and Ui, 
to find an invariant of G separating vg and Vi. An invariant of G, namely the 
polynomial is known from the beginning but the problem is to find one that 
separates vg and vi. 

If there is a algorithm that determines the order any element g G -F in polynomial 
time, then the cryptosystem can be setup and broken in polynomial time, hence 
it is not secure. Even if there is no algorithm that determines the order of g in 
polynomial time, it might be possible to find a separating invariant and break the 
cryptosystem randomly. 

Consider the following example. 

Example 3.1. Let s be a prime dividing q—l,d= G = Gd is generated by g, 
Vg = g and vi = a (a is a primitive element of ). Then the only invariant of G 
that separates vg and vi is x‘^. Therefore breaking of this cryptosystem is equivalent 
to finding of the order of g, and is also equivalent to finding of the prime factor 
s of q — 1. Therefore we conclude that breaking of all cryptosystems of this type 
is equivalent to finding of all prime divisors of q — 1 (hence finding of the prime 
factorization of q — 1.) 

We give a brief review of the computational complexity of factorization of integers 
in the next subsection. 

The value of the above example is in showing that even if an invariant of G is 
known, it might not be completely trivial to find an invariant of G separating vg 
and vi- 

Of course we can find the order of g and break this crytosystem using the discrete 
logarithm (although it might be easier just to find the prime factorization of g — 1). 
Since the multiplicative group of F has a primitive element a, and the multiplicative 
group of G has generator g, we can use the discrete logarithm to find out the 
exponent h such that g = a^. Then g is the primitive d = root of 

unity. Once we know the order d of g, we have found the invariant x‘^, separating 
Vg and vi- 

Nevertheless, we will not use the case n = 1 to setup a cryptosystem due to con¬ 
cerns about its security. The reason is that in the original setup of the system some 
partial information about the order d of g is required. Only a partial information 
about the order d is required to break such a cryptosystem and we are not aware 
of an effective setup when it is easy to create such a cryptosystem and difficult to 
break it. 

3.2. Computational complexity of the factorization of integers. We refer 
to |28j for the description of various algorithms and their complexity. 

First we will overview the determininistic algorithms for factorization of integers. 

One of the simplest is the Fermat algorithm that works fast n = pq is a product 
of two primes that are of the same magnitude. The most popular detemininistic 
algorithms for factorization of integers (all of them of exponential complexity) are 
(p— l)-method, Pollard’s p-method and the Pollard-Strassen algorithm. These algo¬ 
rithms are often used to find small prime factors. For more details and description 
of other algorithms, see Chapter 2 of [28] . 
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When working over finite field F = GF{q), where q = for a prime p, the 
following theorem helps to determine the factorization of g — 1 . 

Theorem 3.2. (Theorem 2.21 of Let b,k G N, b > 1, and n = b^ — 1. If p is 
a prime number dividing n, then one of the following two assertions holds: 

1. p\b'^ — 1 for some d < k, d\k; 

2. p = 1 (mod k). 

If p > 2 and k is odd, then, in the seeond case p=l (mod 2k). 

Although this statement seems easy to use, in reality it gives an algorithm of 
exponential complexity. 

Probabilistic algorithms for factorization of integers with subexponential com¬ 
plexity are discussed in details in Chapter 3 of [28]. The complexity of these al¬ 
gorithms is of the form L„[ 7 ,c], where 7 = ^ or ^ and c is a positive constant, 
where 

— ^(c+o(l))(logx)'''{loglogx)^~'' 

and o(l) —>■ 0 as n —>■ oo. 

Most popular algorithms of this nature are Lenstra eliptic curve method, qua¬ 
dratic sieve and number field sieve. For more details see Chapters 3 and 4 of [28] . 


3.3. n = 2 and G cyclic. Next, we will discuss the case when G is cyclic and 
n = 2 and show that breaking of the corresponding cryptosystem is equivalent to 
solving of a discrete logarithm problem. 

Assume that F = FG{q) is a finite field of cardinality q, where q = p^ and a be 
a primitive element oi F^. 


Let a cyclic group G be generated by the element 7 = 
and 72 = aL and the set S consists of vectors Vi = 


Oil 

012 


y where 71 = 
0 72 / 

for i = 1 ,..., s. 


Since every invariant of G is a sum of monomial invariants, to obtain a complete 
description of all invariants of G we only need to find monomial invariants. A 
monomial / = is an invariant of G if 71 ^^ 72 ^“ is constant for every integer 

X. Plugging in a; = (7 — 1 we get 71 ^ 72 ^ = 1 which is equivalent to 


dih + ^ 2^2 = 0 (mod q — 1 ). 


A monomial invariant / = separates vectors Vi and Vj if and only if 

di do / d\ do 

The following is an example of a group G for which finding an invariant / separat¬ 
ing vectors from the given set S is computationally hard problem - see Proposition 

ISH 


Example 3.3. Assume that F = FG{q) is a finite field of cardinality q, where 
q = p^ and s is a (large) prime that divides q — 1. Let a G F^ be an element of 
order s, and let (d = for a secret integer b not divisible by s. 

Let V = F"^, G C GL{V) be a cyclic group generated by the element g = 

, and the set S consists of vectors Vi = > where ai = a® for i = 

0,..., s — 1. Consider the cryptosystem based on this group G and the set S. 


fa 0 

Vo fd 
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A general element of G, written as for some exponent x, acts on vectors Vi as 


g = 


a 


Wi 

W2 


= w, for i = 0 , 


, s - 1 . 


3.3.1. Discrete logarithm approach. If the exponent b is known, then we can decode 
Oi and Vi from w simply using Oi = ^. 

Otherwise, we can break this cryptosystem if we have an effective algorithm for 
the discrete logarithm problem in . Indeed, to determine i it is enough to solve 
for y such that (P-)y = 

3.3.2. Invariant approach. 

Proposition 3.4. Finding a monomial invariant f of G from Example sepa¬ 
rating elements of S is equivalent to finding b (mod s), hence to finding a solution 
of the discrete logarithm problem for the pair a and /3. 

Proof. We have chosen vectors Vi in such way that none of the invariants of G of the 
form a;f or a;| would separate any two of them. Since there is a monomial invariant 
of G separating all elements of the set S (for example x^x^), we can assume that 
it is of the form / = xf^x^^, where di,d 2 ^ 0. Then 

di + bd 2 = 0 (mod s) and whenever i j. 

This implies f{w) = and that 

determines Vi. 

Therefore finding a monomial invariant / separating elements of S requires find¬ 
ing a solution of the congruence di bd 2 = 0 (mod s) which is equivalent to deter¬ 
mining b (mod s) and this is equivalent to the discrete logarithm problem for the 
pair a and p. 

Conversely, if we are able to determine b (mod s), then fr = x^xf^ is a rational 
invariant of G and x\x 2 is a monomial invariant of G separating all elements of 
S. □ 

Let us describe an algorithm how to describe all invariants of cyclic group G 
when n = 2 and find one separating elements of S. Finding a monomial invariant 
Xi^xY of G is equivalent to solving the congruence dpi -\-d 2 l 2 = 0 (mod q— 1). To 
find such an invariant, we first need to find the primitive element a and use discrete 
logarithms to solve for li and I 2 from 71 = and 72 = 

Once h and I 2 are known, there is an effective way to describe all solutions of the 
above congruence and all monomial invariants of G as follows. If GCD{li, q—1) = 1, 
then we can choose any d 2 and compute di = —If^d 2 l 2 (mod q — 1). Here we find 
If^ (mod 9 — I) using the Euclidean algorithm which runs in polynomial time. If 
GCD{li,q-l) = d> I, then d|d 2 ^ 2 , We replace h by I 2 by = GCD(h d) ’ *^2 by 
d '2 = and the congruence di/i+^ 2/2 = 0 (mod q—1) by + = 0 

(mod The solutions of the last congruence are described analogously as above 

since GGD{1[, = 1. Finally, we compute d 2 = d '2 gcd {12 d) • ^i^^e we know all 

invariants, it remains to select one separating elements of S. 
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3.3.3. Complexity and expansion in size. Coming back to the Proposition 13.41 we 
have seen that even in the simplest case when n = 2 and the group G is cyclic the 
task of finding invariants of G separating elements of S is of the same complexity 
as the discrete logarithm problem. 

Since the discrete logarithm assumption is weaker than the computational Diffie- 
Hellman assumption and that is weaker than decisional Diffie-Hellman assumption 
which are used in Diffie-Hellman key exchange and ElGamal encryption, we have 
a guarantee that the invariant-based cryptosystem of Example 13.31 is at least as 
secure as many standard and widely used public-key cryptosystems. 

If we compare the cryptosystem from Example 13.31 to ElGamal public-key cryp¬ 
tosystem, we note that while ElGamal encryption produces a 2:1 expansion in size 
from plaintext (by this we mean a sequence of O’s and I’s) to ciphertext, and the 
invariant-based cryptosystem from the above example produces 2 log 2 q : log 2 s ex¬ 
pansion in size from plaintext to ciphertext. If s is not small in comparison to 
q, say ^ is close to 1 , then this expansion will be close to 2 : 1 . Also, higher 
the number s becomes, the ratio of expansion from the plaintext to the ciphertext 
would improve. 

In general, if s is small or if it is comparable to q, then finding the prime s 
would be easy. If g — 1 = S 1 S 2 S 3 , where s = si 7 ^ S 2 are two large primes and S 3 is 
small integer, then it could be challenging to determine the order s of the element 
a since one needs to determine the prime factorization of g — 1. In this cases the 
best choice would be if ^ is close to This implies that the corresponding 
expansion from plaintext to ciphertext will be close to 4:1 (Perhaps not a big price 
to pay for increasing the security of the cryptosystem). 

In the case of general n and a set S of cardinality r we get nlog 2 \F\ : log 2 r 
expansion. A trivial upper bound on the cardinality r of the set S is given by the 
index [A” : G]. Adjusting the previous example for n = 2 we are able to get the 
expansion ratio close to n : 1 - and quite possibly even better ratio with different 
choices of S. While such ratio is a disadvantage, if we choose n small, this should 
not play a big role for the effectiveness of the cryptosystem. 

3.4. Computational complexity of the discrete logarithm problem. Algo¬ 
rithms for computing discrete logarithm are subject of Ghapter 5 of [55]. There are 
deterministic algorithms of exponential complexity and probabilistic algorithms of 
subexponential complexity. There is an algorithms for discrete logarithm problems 
in prime fields of complexity Lp[i;c]. ElGamal [9] gave an algorithm that works 
over Galois fields GF{q) and has the complexity Lq[F,c]. It is interesting that it 
uses a representation of GF{q) as the residue class ring 2’/tp, where Z is the ring 
of algebraic integers of the cyclotomic field Q(Cp), and tp is a prime ideal of Z of 
norm g. 

Another algorithm working in prime fields based on number field sieve has com¬ 
plexity Tp[i;c]. Finally, in section 5.6 of [55] there is an interesting algorithm for 
discrete logarithm with composite modulus based on Fermat quotients that works 
in residue class rings Z/mZ with composite m. 

3.5. G cyclic. When n > 2, the problem of finding monomial invariants of a cyclic 
group G over a finite field F is essentially reduced to multiple applications of the 
method already used to find invariants of cyclic G in the case n = 2. 
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Assume that F = FG{q) is a finite field of cardinality q, where q = p'" and a be 
a primitive element of 

fli 0 ... 0\ 

Let a cyclic group G be generated by the element g = 


0 72 


0 


\ 0 0 


7n/ 


where jj = and 0 < /^ < g — 1 for j = 1,..., n and the set S consists of vectors 

f 

for 1 = 1, 


= 


aa 


,s. 


\^in j 


Since every invariant of C? is a sum of monomial invariants, to obtain a complete 
description of all invariants of G we only need to find monomial invariants. A 
monomial / = ... x'^ is an invariant of G if ... 7 ^"^ is constant for 

every integer x. Plugging in a; = g — 1 we get ■ ■ ■ 7 ^" = 1 which is equivalent 

to 

dill + (^ 2^2 + ... + dnln = 0 (mod q — 1). 

In particular, x^^ x~^' for i ^ j are rational invariants of G and x^^ x^^ 
monomial invariants of G. 

A monomial invariant / = Xi^X 2 ^ ... separates vectors Vi and Vj if and only 
if 


q—l — li 
^3 


are 


d\ d'> drt j d-] I 

^i2 • ■ ■ ^in ^ 


d2 dn 

jl'^j2 ■ ■ 


Proposition 3.5. Let G be a cyclie group, Vk ^ vi be elements of F^ and n >2. If 
there is a monomial invariant f of G separating Vk and vi, then there is a monomial 
invariant of G the form x’l'xf that also separates Vk and vi. 

Proof. We will proceed by induction on n. The statement is trivial for n = 2. 
Assume that the statement is true for all k < n. If we write / = .. .xj^", then 

/ invariant separating Vk and vi implies 

dill + ... + dJn = 0 (mod q-l) and a'l\... ^ a'^l ■ ■ ■ ■ 

Denote d = GGD{li, ..., Z„, g—1). Then there is an index t such that {a'l\ ... a‘1^)^ ^ 


{aff ... o '}") <1. If the invariant Xy 




“it" which gives a 

^. h d — 

” implies 


ku 


L ~T 

= a 


Xff does not separate Vk and vi , then = 


lu 


This together with 






^In 


di^ 0 dt^-\-du^ dn^ , di^ 0 dt^-\-du^ d 


■ • ■ F “Zl 

Replace the space F” by the group G with a group G' generated by the 

matrix g' obtained from the matrix g generating G by deleting its rz-th row and 
column, and replace vectors vt with v[ obtained by deleting the entry in their u-th 


row. By the above, the monomial f' = x^ x^ ... x^'^ 
v'j. and ?;(. Additionally, /' is an invariant of G' because 

di—Zi + ... -I- Oly, + ... -I- {dt— + du—)lt + ... + dn—ln = 

— dill + ... + 0 -|- ... + {dth + duly,) + ... -|- dnln = 0 (mod q — 1). 
d L J 


separates 
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Using the inductive assumption we get an invariant of G' of the form 
that separates v'f. and v'l- It is clear that this is also an invariant of G and that it 
separates Vk and vi. □ 

As a consequence of the above proposition we conclude that in order to find an 
invariant monomial of G separating vectors Vk and vi of S we can proceed by going 
through all the pairs of indices i and j from 1 through n and and solve analogous 

problem when G is replaces by Gij generated by 2 x 2 matrix gij = | T* | and 


vectors Vk and vi are replaced by vector 


^ki 

^kj 


and 


0 7 ^ 

, respectively. 


3.6. G not cyclic. Noncyclic groups G exist for every n > 2. We start with an 
example generalizing Example 13.31 

Example 3.6. Let F = FG{q), where q = p’', and si and S 2 be (large) primes 
dividing q — 1. Let ol\ € F'^ be an element of order si and 02 € F^ be an element 
of order S 2 , and ai be distinct elements of F^ such that their order divides siS 2 - Let 


G be given by two generators gi = 


and (32 = secret integers bi not divisible by si, and 62 not divisible by S 2 Let 

Consider the cryptosystem based on this 


ai 

0 


0 


and (?2 = 


02 

0 


0 

^2 


, where /3i = 


the set S consist of vectors Vi = 
group G and set S. 


ai 


The general element g of G, written as gi^g^^, acts on Vi as gvi = 


Wi 

W2 


= w. 


As before, we have chosen vectors Vi in such way that none of the invariants of G 
of the form or would separate them. If there is an invariant of G separating 
vectors Vi, then we can assume that it is of the form / = 

If / = x'^x')^ is an invariant of G, then (/ 3 f^ = 1 . If a is the 

primitive element of ai = and 02 = for secret integers ai and 02, then 
this is equivalent to 

(aixi + a2X2)di + (aibiXi + a2b2X2)d2 = 0 (mod 9 — 1) 

for every xi and X2- This condition is equivalent to the system of congruences 

ai{di + bid2) = 0 (mod 9 — 1) and 02(^1 + 62^2) = 0 (mod 9 — 1). 

These two congruences are related to different generators gi and 92 of G. We have 
seen earlier that using discrete logarithms we can describe all monomial invariants 
of the cyclic subgroups (91) and (92) of G. Of course, the invariants of G are exactly 
those polynomials that are invariants with respect to (91) and (92) simultaneously. 

The last two congruences can be solved by using integer linear programming 
because they are equivalent to the linear system 

aidi+ ai 6 ic ?2 + (9 — I)d3 = 0 
a2di+ 0262^2 + (9 — I)c!4 = 0 

in integers di, d2, da and d4. 


PUBLIC-KEY CRYPTOSYSTEM BASED ON INVARIANTS 


11 


(ill 0 

0 Iii 


0 


Now consider the general case when G has generators gi = 

\ 0 0 ... lin) 

for i = 1,.. . t, where m and 0 < ly < g — 1 for j = 1, ..., n and the set S 

( 

for * = 1,..., s. The general element of G is written 


consists of vectors Vi = 


aa 


\^in J 


as g = g\^ ... gf* for some integers yi,... ,yt. 

Since every invariant of C? is a sum of monomial invariants, to obtain a complete 
description of all invariants of G we only need to find monomial invariants. A 
monomial / = ... xfl is an invariant of G if 111=1 lli^^li 2 ^^ ■ ■ ■ 

all integers yi,... ,yt. This implies if^ 1(2 ■ ■ ■ itn = 1 for each i = 1,..., t which is 
equivalent to the system of congruences 

+ ^ 2^12 + ... + dnlin = 0 (mod g — 1) 


for each i = 1,..., t. 

If the numbers are determined (say using discrete logarithms), then the last 
system can be solved using integer programming since it is equivalent to the system 

dihi + ^2^12 + . •. + dnhn + d„+i(g — 1) = 0 


for each i = 1,..., t in integer variables di,..., 


A monomial invariant / = Xi^X 2 ^ 


. xfl separates vectors u) and Vj if and only 


if 




d 


di d2 




We have seen that, for F a finite held, the fact that is cyclic allows the use 
of the discrete logarithm, which is computationally difficult but standard crypto¬ 
graphic tool. 

If G was cyclic it was enough to hnd any invariant of G randomly and check if it 
separates S. If G is not cyclic then more systematic knowledge of invariants of cyclic 
subgroups of G is necessary and breaking of the cryptosystem based on noncyclic 
group G seems more complicated than the case of the cyclic group G. Additionally, 
it is not clear if there is a separating invariant based on two variables analogous as 
in Proposition I3.5I in the case of noncyclic group G. Therefore using noncyclic G 
gives an advantage from the point of view of the security of the cryptosystem based 
on invariants of G. 

The setup will be even more complicated if the underlying structure of F is not 
cyclic. After we investigate the minimal degree of polynomial invariants (question 
related to a linear algebra attack on the cryptosystem) in the next section, we turn 
our attention to fields F of characteristic zero. We will introduce a cryptosystem 
the security of which will depend on the fastorization properties in the number field 
F. Afterward, we will use residue classes of F and replace T by a finite commutative 
ring R with a complicated multiplicative structure that will not allow an obvious 
use of the discrete logarithms. In the cases of the number field F and the residue 
ring R, the nature of finding invariants of G is different from discrete logarithm 
problem. This is clear in the case of a number field F. The residue residue ring R 
have divisors of zeros and the multiplicative group U of its units is not cyclic. 
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3.6.1. Encryption based on discrete logarithm one-way functions. We would like to 
make a small detour from the invariant-based cryptosytems and discuss cryptosys¬ 
tems based on discrete logarithms that are inspired by Examples 13.31 and 13.61 

Assume that Alice chooses a finitely generated group G acting on a set M. Let 
{gi, ..., Pm} be a set of generators of G. Alice chooses a subset Mq C M such that 
any orbit Gm for m G M intersects Mq in exactly one point. (The set of all blocks 
of plaintext that can be transmitted by Bob is injectively mapped to Mq.) Alice 
chooses a map f : M ^ Mq that is constant on each orbit Gm of G and retains it 
as a private key. Obviously / restricted on Mq is an identity. She announces, as a 
public key, the (effectively described) set Mq and the group G, by announcing its 
generators gi,. ..,gm. 

To encode a block of plaintext m G Mq, Bob choses a random element g G G 
(by multiplying some of the generators gi,..., gm), and computes m' = gm which 
he transmits to Alice. 

Alice decrypts the message by applying the map / as f(rn') = f{gm) = f{m) = 

m. 

Example 3.7. Consider the ElGamal cryptosystem with cyclic group G of order 

n, generator a G C, private key b G {0,1..., n — 1} and the public key {a, jd, n}, 
where fi = of’ . The group G coincides with the set of all blocs of plaintext that can 
be send by Bob to Alice. A cryptosystem is contracted as follows. Let M = G x G, 
considered as a group with respect to the mutiplication induced by the diagonal action 
of C, and let G to be its cyclic subgroup generated by {a, (3). Then G acts on M by 
multiplication. We set Mq = G and the map f : M ^ Mq to be f{x, y) = yx~^. 

Example 3.8. Let A be an abelian group generated by Q;i,a 2 G A. Let /? = 
for some &i ,&2 G N. Let M = A x A x A and G be its cyclic subgroup generated 
by the element {ai,a 2 , (3). We set Mq = A and the map f : M ^ Mq to be 
f{x,y,z) = 

Alice announces the group A and the vector (ai,a2,l3) as a public key. To encode 
a block of plaintext m G A, Bob chooses a random number e G {0,1,..., n — 1} 
and transmits the vector (m, ^ 2 ,/3)®(1,1, m) = (a®, q;|,/3®to) to Alice. 

Alice decrypts the message m as f{al, a^, P'^m) = f3^maf'^^^ = m. 

This encryption produces a 3:1 expansion in size from plaintext to ciphertext. 

The security of this cryptosystem depends on the ability of the evesdropper 
Charlie to solve the equation [3 = for integers xi and X 2 . If we work over 

a finite field F = FG{q), then we can we can use the discrete logarithms in to 
express ai, a 2 and P as powers of the primitive element a of F^, say ai = a®* and 
P = a®. Then the equation P = reduces to a congruence e = eiXi -I- 622:2 

(mod q — 1). Therefore we require the discrete logarithm assumption to guard 
against this attack. 

4. Minimal degree of polynomial invariants of G 

When we described the design of the cryptosystem based on invariants we have 
already remarked that its security depends on the difficulty of finding an invariant 
/' of the group H separating vectors in S. 

When we are working over a ground field F of characteristic zero, then the 
condition that f' separates Vi and Vj for i ^ j might not be difficult to satisfy 
because the set of polynomials in E[E], that take on different values when evaluated 
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at Vi and Vj, is open in the Zariski topology. Therefore it is likely that a randomly 
chosen invariant f' of H will separate elements of S in this case. Therefore when 
F has characteristic zero, we need not be concerned whether f separates vectors 
from S. 

This is contrary to the situation over finite fields when it was easy to find an 
invariant of G but difficult to satisfy the condition that it separates elements of S. 

4.1. Guarding against the linear algebra attack. Denote by Mcy^ or simply 
by Mg or M if we need not emphasise the group G or the vector space V it 
is acting on the minimal positive degree of an invariant from F[V]^. That is 
May = min{d > 0|F[F]^ ^ 0}. If F\V\^ = F, then we set May = oo. 

The notion of the minimal positive degree of an invariant and the value of 
M = May are important for the security of the invariant-based cryptosystem 
(both variants one and two) we are considering. For example, if we know that 
Mg is so small that “ 0{rF) is polynomial in n, then Charlie can find 

an invariant /' of G in polynomial time by solving consecutive linear systems for 
d = 1,..., each consisting of equations in the vari¬ 

ables described in the previous section. For a fixed d, this can be accomplished 
in time and the total search will take no more than time 0(n®’'). 

Therefore, for the security of the system it must be guaranteed that is 

not polynomial in n. 

4.2. Finding a polynomial invariant of G. We will now discuss an algorithm 
that will enable us to find an invariant f of G (and to break the cryptosystem if 
char F is zero). The algorithm works inductively, and as a special case, it works 
when G is a finite group. We will apply this algorithm when char F is zero but it 
works even when the characteristic of F is finite. 

Assume that i? is a subgroup of G of finite index in G. Assuming we know a 
nonzero invariant / of iJ, we will find a nonzero invariant of G. 

Lemma 4.1. Let FI be a subgroup of G of finite index s in G such that f is an 
invariant of Ft of degree t. Then G has a nonzero invariant of degree not exceeding 
sMh that can be found in time 0(sn*“''^("~'’*~^) ). 

Proof. Denote by gi,... ,gs, where s = [G : Ft], representatives of all coset classes 
of G/Fl. Let / be an invariant of FI of degree Mh- Denote Xi = gif for z = 1,... s, 
and denote by Ps{xi,... ,Xs) = xi .. .Xg the s-th elementary symmetric function 
in xi,..., Xg. It is easy to see that Pg(xi,... ,Xs) is invariant with respect to G, 
because each element g G G permutes coset classes of G/Fl, hence it permutes 
the set of polynomials {xi, ...,a:s}. Also, the polynomial pg(xi, ...,cc^) = xi .. .Xg 
is nonzero and has the degree sMh- We can evaluate all polynomials Xi in time 
0{sn^{^^\~^)n*'). The product of all Xi can be computed in time 0(("''’*“^)*). □ 

Corollary 4.2. If G is a group of finite order s, then the algorithm in the proof of 
the previous lemma (applied to H = 1) produces a nonzero invariant of G of order 
not exceeding s which can be computed in time 0(sn^n'’+^). 

Note that the time required to run the computation is exponential in the order 
of G if no invariant of a subgroup of G is known and when we attempt to find an 
invariant of G from H = 1. Nevertheless, there are cases when an invariant of H 
can be computed in polynomial time; see the next lemma. 
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The following lemma is well-known, see [5]. 

Lemma 4.3. If G G G'L„(M) and G is finite, then G has an invariant of degree 
two. 

Proof. Let gi = 1,... ,gs be all elements of G and K[y] = ... ,tn]- Denote 

by = gfit^ + ... + t'^) for i = 1,..., s. Since values of each Xi are non-negative 
when evaluated as polynomials in ti,..., the values of the invariant polynomial 
Si=i evaluated as polynomial in are non-negative and they can be 

equal to zero only if each Xi is zero. But Xi = 0 only if = ... = = 0. Therefore 

Si=i positive-definite quadratic form in ti,..., hence a non-zero invariant 

of G. □ 

It follows from the previous section that a quadratic invariant of the group 
II, within a context of our public-key cryptosystem, can be found using linear 
algebra techniques in the polynomial time in n. Therefore, for the security of the 
cryptosystem, we need to make sure that if If is finite, then it is not represented 
by matrices with real coefficients. 

4.3. Lower bounds for degrees of polynomial invariants. The significance 
of understanding the minimal degree Mq v of invariants for the security of the 
invariant-based cryptosystem was established above. In particular, it is important 
to find a nontrivial lower bound for May. Unfortunately, we are not aware of any 
articles establishing lower bounds for the minimal degree of invariants, except in 
very special circumstances, e.g. m- 

On the other hand, there are numerous upper bounds for the minimal degree 
/3(G, V) such that F[V]^ is generated as an algebra by all invariants in degrees not 
exceeding /3(G, V). For example, a classical result of Noether [20] states that if the 
characteristic of F is zero and G is finite of order |G|, then /3(G, V) < |G|. There 
is an extensive discussion of Noether bound and results about /?(G, V) in section 3 
of |2l]. It was conjectured by Kemper that for G 1, and arbitratry ground field 
F, the number fi(G,V) is at most dimU(|G| — 1). Recently, this conjecture was 
proved by Symonds in [26] . 

When one wants to find an invariant of G, it seems natural to consider an upper 
bound /3(G, V). However, if we wants to show that there are no invariants of small 
degrees (as is our case), then we need to find lower bounds for May. Until now, 
there was no real impetus to consider such problem. We have investigated minimal 
degree of invariants of G in general in |15j . where we have obtained its description 
for certain groups G. 

5. Cryptosystems based on invariants of infinite diagonalizable 

GROUPS 

In this section we assume that the characteristic of the ground field F is zero 
and we design and ivestigate the properties of a cryptosystem based on invariants 
of an infinite diagonalizable group. 

Let us fix a number field F = Q(0) and the subring Z = T\0] of the ring of 
algebraic integers of Q(0). Choose a finite set Q of integers of cardinality q and 
a set Sm = {pi,. ■. ,Pm} of elements of Z. The elements pi of Sm need not be 
primitive and could be units of Z. Denote by Pm the set of all products of elements 
from the set Sm- 
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5.1. Design of the cryptosystems. To start, Alice chooses sets Q and Sm as 
above. Afterwards, she chooses her secret key, which is the n-tuple of nonnegative 
integers (ei,..., e„), where one component, say e„ equals 1. Then she will construct 
a set of generators ti,... ,ts of T in such a way that the monomial f = ... x®" 

is invariant under the action of each ti, hence belongs to F[V]’^. 

At the i-th step of the process, Alice chooses the i-th generator ti of the group 
T as follows. 


First, for every k = l,...,m and 1 < j < n — 1 she chooses numbers bj. 


(i) 


from the set Q. Then she computes the numbers 


(i) 


= rifeLi Pk'^ where 1 < j < n — 1. Alice then computes aJC"* in such a way that 
the diagonal matrix gi = diag{a^i\... ,an'^) has / = x^^ as an invariant. 

Since she has chosen e„ = 1, it is easy to see that the appropriate value of is 

Once all generators ti of the group T are constructed, Alice chooses an invertible 
n X n matrix P as a part of her secret key and computes conjugates gi = PtiP~^. 
Alice then announces the diagonalizable group G given by its generators gi for 




from the set Prr 
b) 


J 

as 


i = l,...,s. 

When she receives the encrypted message, she can use her secret key P to switch 
from G to T and apply her previously chosen invariant / = x^^ ... cc®" of T to 
decrypt the message, as explained in section [2j She knows that / is an invariant of 
T because T was constructed to satisfy that condition. 

To remove the randomnes of the choice made during this process, Alice should 
use a cryptographically secure pseudorandom number generator. 


5.2. How to break the cryptosystems in partial cases. We will explain how 
the above cryptosystem could be broken in the polynomial time for some rings Z. 

Lemma 5.1. Assume that a ring Z is such that the group of units of Z is finite, 
Z is an Euclidean domain, and the Euclidean algorithm over Z runs in polynomial 
time in its input. If a vector, encrypted by the above cryptosystem, has no zero 
components, then it can be decrypted in polynomial time. 

Proof. If the group of units of Z is finite, then it consists of roots of unity. Assume 
that it is generated by Cs- 

At first, we compute the characteristic polynomials of all matrices gi and find all 
of their eigenvalues. This can be done in polynomial time using the algorithm for 
factoring a polynomial over a number field described in Section 3.6.2. of [6]. Hence 
the factorization of all characteristic polynomials can be done in polynomial time 
in n. Then, we follow the algorithm explained in the proof of Proposition 15.4 of 
m and simultaneously diagonalize all matrices gi and obtain generators ti of the 
conjugate group T' consisting of diagonal matrices. For simplicity of notation, we 
can assume that T' = T. Actually, what is important for us are only the eigenvalues 
..., On^ and their order with respect to fixed order of the eigenvectors. We will 
not work with the actual eigenvectors of V. 

Since all eigenvalues are ratios of elements from Z, we can consider the set X 
of integers that appear in the numerators or denominators of any eigenvalue of any 
matrix ti. Using Euclidean algorithm we can compute the set Y of all greatest 
common divisors of all pairs of elements from X. Then we can write a partial 
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factorization of all elements of X in the form where x = yz and y is a product 
of elements from Y and Z is not divisible by any element from Y. Afterward we 
replace A by a new set X' consisting of all elements in Y and of elements z from the 
above factorization. In the next step we replace the set Y by the set Y' consisting 
of all greatest common divisors of all pairs of elements from X'. We continue in 
the same fashion and after finitely many steps this process will stabilize. Then we 
arrive at a set of numbers that are pairwise coprime divisors of integers from 
X. Let us call elements of atoms of X and denote them by {ai,... am'}- Since 
the Euclidean algorithm in Z runs in polynomial time in n and there are no more 
than qm steps of the above process, we find the atoms in polynomial time in n. 

For every atom a, every element x of A is either coprime to a, or is writ¬ 
ten as X = a^b, where b is coprime to a. Every has the atom factorization 

< E. We can hnd an invariant of T from 


(b 


— rr™ 

— SE llfe= 


b'}'] 


k=l “fc 


where 0 < Cij 


the structure of these diagonal matrices by solving, in nonnegative integers, the 
system of s equations = 1 in n variables. Since Charlie has the atom 

he can compare the exponents in the atom fac- 


factorization of each element 
torization and obtain a system of s{m' -I- 1) linear equations ~ ^ and 

— 0 with bounded coefficients and Cij in n variables yj. An integer 
solution of this system can be found in polynomial time in n - see subsection 1.5.2 

ofH. 

The task to find a nonnegative integer solution of the linear system with integer 
coefficients is an NP-complete problem. If a solution of our system that has all 
nonnegative components is found, then it corresponds to a polynomial invariant of 
G. 

However, every integral solution corresponds to a rational invariant, that is a 
rational function that is invariant under the G-action. If the intercepted encoded 
vector has no zero coordinates, then Charlie can use his rational invariants to decode 
the message. □ 


Let us remark that the assumption of the above lemma are satished for integers 
Z = Z or Gaussian integers Z = Z[i]. It is well known that the Euclidean algorithm 
runs in polynomial time over Z and Z[i]. For a survey of algorithmic results see 
Section 3 of [1]. Also, the above results can be extended further if we replace 
the assumption that Z is Euclidean domain by the assumption that Z is complex 
quadratic unique factorization domain. According to [16] there is an algorithm, 
running in polynomial time, that computes gcd in such rings Z. 


5.3. Theory of divisibility and units in algebraic number fields. Assume 
that E is a number field, that is a finite extension of the field Q of rational numbers. 
Let Z be a ring of algebraic integers of F. In many rings Z factorization of elements 
into primes is not unique (hence Z are not unique factorization domains), see for 
example the case when F = Q(-^—5) in 2.3 of [3]. 

Recall the theory of divisibility essentially due to Kummer from Section 3 of [3] . 
We replace an element a € Z by the principal ideal (a) of Z generated by a. The 
ring Z is a Dedekind domain and it has a theory of divisors. I particular, each ideal 
a of Z can be written uniquely as a product a = pi... pr, where pi for i = 1,... r 
are prime ideals of Z. The ideals a and b of Z are called equivalent if there exists 
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a G such that a = b(a). The set of all equivalency classes is called the class 
group of F. It is a finite group of order h, called the class number of F. The ring 
Z is a unique factorization domain if and only if /i = 1. 

In relation to factorization of elements from Z it is important to recall the 
structure of the group U of units of Z. By the Dirichlet theorem, if r is the number 
of real embeddings of F and s is the number of complex embeddings of F, then the 
group is isomorphic to a product of a finite group of roots of unity and a free group 
of rank r + s — I, whose generators are called fundamental units of Z. 

5.4. Security issue - the choice of the ring Z. The choice of the ring Z is 
perhaps the most critical since the security of the cryptosystem depends heavily on 
the arithmetic of the ring Z. 

The atom or prime factorisation analogous to the one considered in the proof 
of Lemma 15.11 is not available in suitable form for number fields in general. For 
simplicity assume that Z = ^[6] coincides with the ring of algebraic integers of the 
field Q(ff). If .Z is a principal ideal domain but not a Euclidean domain, then we have 
a factorization of every element of Z into a product of primitive elements and units 
of Z. However, without the Euclidean algorithm, it is not clear if we can produce 
prime factorization of principal ideal in polynomial time. If Z is not a principal ideal 
domain, then instead of primitive elements we need to work with divisors. Namely, 
for each x ^ Z there is the prime ideal decomposition [x) = pi .. .p/, where pi are 
(not necessarily principal) ideals in Z. An ideal generated by each prime number 
p splits up to a product of many prime ideals (their number does not exceed the 
degree of the extension [Q(0) : Q], and this number is attained for totally ramified 
primes p). The problem of finding the prime ideal factorization in Z is very difficult. 
Its special case for F = <Q is the prime factorization problem in Z. The difficulty 
of factoring of a product of two large primes is the basis of the RSA public-key 
cryptosystem. We should consider only those rings Z = Z[9] for which their class 
number is bigger than one. Such ring Z is not a unique factorization domain (and 
consequently not a principal ideal domain and not an Euclidean domain). 

Even if we assume that the prime factorization of principal ideals generated by 
is known, by itself it would not be enough to break the above system. The 
additional difficulty lies in the structure of the group of units of Z. For example, if 
we choose all elements of Sm to be units of Z, then the whole idea of atom or prime 
decomposition is utterly useless. In order to facilitate the conversion into a system 
of linear equations, we would need to determine a factorization of each appearing 
unit into a product of roots or unity and fundamental units of the ring Z. Finding 
a set of fundamental units of the ring Z and decomposition of units of Z into 
products of root of unity and fundamental units is by itself a very difficult problem 
and we are not aware of any algorithm solving these problems in polynomial time. 
Therefore the break described in Lemma ISTTl cannot be duplicated for rings Z that 
are not unique factorization domains or those containing units of infinite orders. 
We remark that there is a plethora of examples of such rings Z appearing in the 
algebraic number theory. 

A combination of obstacles related to factorization of principal ideals and factor¬ 
ization of units of .Z as a product of fundamental units is the reason why we propose 
the above cryptosystem based on approprately selected Z. We are unable to find a 
polynomial algorithm for finding an invariant of the corresponding diagonalizable 
group G. 
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To summarize, we need to choose the ring Z in such a way that it is not a unique 
factorization domain and preferably such that its class number is high. There are 
numerous examples of rings of integers of number helds that satisfy this condition. 
Secondly, we should choose Z so that the rank of its group of units is high. Using 
Dirichlet theorem, this condition is easy to satisfy. 

5.5. Other security issues. We will consider other possible choices Alice can 
make and how they affect the security of the system. The additional choices that 
affect the security of the crytosystem (besides the choice of Z) are the following. 

5.5.1. The choice of the set Sm- We could choose elements pi from the set Sm in 
such a way that some of them are primitive. Also, we should choose them in such 
a way their norms will have many common prime factors p. If we chose them 
randomly, then there is a great probability that the prime ideals dividing p in the 
prime decomposition of different Pi are actually different. Also, we could choose 
some elements of Sm to be units of Z in order to involve the structure of units of 
Z. 

5.5.2. Choice of the set Q. A choice of a finite set Q does not seem to be important 
hence we can take it to be small, for example Q = {—1, 0,1}. 

5.5.3. The choice of the secret key (ei,..., e™). Another important requirement we 
need to impose is that none of the entries (ei,..., Cn) vanishes. The reason for this 
is to guarantee that the invariant / we chose depends on all the variables xi,..., x„. 
While we cannot guarantee that there are no invariants of G built on fewer than 
n variables, chosing our invariant / that depends on all variables is a reasonable 
precaution. When we increase the number of generators U of T, it is more likely 
that T would not have invariants depending on small number of variables. A more 
careful analysis of this relationship would be desirable. 

In order to prevent linear algebra attacks described in Section HTTl the secret key 
(ei,..., e„) must be chosen so that E = is least of the order of n. For 

example she can choose Ci € {1,2} such that e* = [^]- also 15.61 a') below. 

5.5.4. Choice of the exponents 5}}. We would like to make sure that the minimal 
degree Mt is close to E, which is the degree of /, or at least of the order n. 
However, if the number s of generators t is high and all exponents are chosen 
randomly, we expect that Mt is going to be of order n. It is an interesting problem 
to investigate how to choose 6^*^ to guarantee that Mt is sufficiently large, say 
bigger than E/2. 

If we cannot gurantee that Mt is of order n, then we can add another generator 
diag{fE, ■ ■ •, (e) to T. That would require replacing the field Q(0) by Q{6, (e) and 
chainging the ring Z. 

This would give away to Charlie the degree of our invariant / but it would also 
make sure that Mt = E. Since E is of order n, this prevents the linear algebra 
break discussed in subsection 14. II 

5.5.5. The choice of the transition matrix P. The idea of using conjugate group 
G instead of T is to make matrices representing elements g G G as far away from 
the diagonal matrices as possible. Therefore the matrix P should be complicated, 
and with many nonzero entries, in order to accomplish this. Please see the next 
subsection 15.61 part b) about the security of conjugation by P. 
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5.6. Possible attacks. We will now describe possible attacks on the above cryp¬ 
tosystem. 

a) Linear algebra attack 

Charlie might attempt to find an invariant of G directly using the linear algebra 
attack described in Section [4.II The complexity of this approach is exponential in n 
if Me is of the order of n, which is likely going to be the case due to (random) choices 
of and which can be guaranteed by adding another generator diag{Cn, ■ ■ ■, Cn) 
to T. Therefore this linear algebra attack is ineffective. 

b) Finding the conjugate group T 

Charlie might attempt to hnd a conjugate group T' of G, consisting of diagonal 
matrices. In order to diagonalize G, he would find all eigenvalues of elements gj by 
computing their characteristic polynomials, which he can do in polynomial time in 
n. There exists a polynomial algorithm for factoring a polynomial over a number 
held - it is described in Section 3.6.2. of [5]. Hence the factorization of all charac¬ 
teristic polynomials can be done in polynomial time in n. Once the eigenvalues of 
matrices corresponding to every gi are computed, he can simultaneously diagonalize 
all matrices gi (see the proof of Proposition 15.4 of [13]) and obtain the generators 
of a group T', in polynomial time in n. 

This suggests that the conjugation by P, suggested in [10] as a way of ’’hiding” 
the group G and its invariants, is not secure without our context. 

c) Finding an invariant using ideal and units factorisation 

For rings Z, that are non-Euclidean or have inhnite group of units, the attack 
described in Lemma l5.1l is not viable. 

5.7. Possible modification of the system. We have seen before that switching 

from the system of equations = 1 to the linear system ^k^Vj = 0 

is important for possible breaking of the system. This can be accomplished by prime 
ideal factorization - see 15.61 c) above. One possibility to prevent this method is to 
choose the numbers to be arbitrary and random complex numbers. Then the 

corresponding linear system would consists of equations 

appears to be difficult to find a solution of such general system in integers. 

On the other hand for computational purposes we need to approximate the num- 

(i) 

bers a) by complex numbers with finite decimal expansions. This would create 
difficulty estimating errors of the encryption process. For such system it would 
be necessary to estimate possible error of encryption and also it would be neces¬ 
sary that the vectors Vi from the set ^used in the encryption process could be 
distinguishable within the errors of such computations. 

5.8. More general systems considered in [11]. The main reason we were able 
to design a system for diagonalizable groups was that we were able to easily con¬ 
struct matrices that have a given monomial as its invariants. In the case of hnite 
diagonalizable G, a reasonable description of the invariants for diagonal matrices 
is given in [TS]. For inhnite diagonalizable G the situation is similar but we have 
equations instead of congruences. 

One could hope that designing a system based on nonabelian G would be more 
secure than that based on a diagonalizable group T because it is more complicated 
to hnd invariants of such G than those of T. A system based on invariants of 
nonabelian group G would have an advantage that simultaneous diagonalization 
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as described in 15.61 b) is not possible. Therefore the conjugation problem is more 
difficult to solve for nonabelian G. Also, we need to take into account that the 
minimal degree of G must be at least of order n to prevent linear algebra attacks. 

In the paper m the authors have proposed a process of generating a more 
complicated (nonabelian) group G, its representation and a corresponding invariant 
starting from simpler groups using four types of operations. Their main idea was 
that it would be more difficult to hnd an invariant of G than that of the simpler 
groups. We will investigate how this construction affects the minimal degrees of 
invariants since they are important in regard to the possible linear algebra attack 
on the corresponding cryptosystem described in subsection 14.II 

For the first operation, assume that G < GL{V), where V ~ i?" is a free 
module over a ring R of rank n; and a ring homomorphism tt : i? —^ R' , replacing 
R with a new ring i?', are given. If R' is a direct summand of R and tt is a 
projection onto R' (in which case R' is called smaller), then every invariant of 
R[V]'^ remains an invariant of i?'[7r(I^)]‘^, hence this operation does not increase 
the minimal degree May- If R is embedded into R' , then C R'[R' (g)/j I^]*^, 

hence Mcy^^v < May- The authors of [11] do not specify what they mean 
when R' is larger, and we were unable to follow their arguments. However, if the 
kernel of the map tt is nontrivial, then some of the invariants can be annihilated 
using this process and the minimal degree can potentially increase. 

The second operation replaces G by a conjugated subgroup H = h~^Gh for some 
h G GL{V). Since the algebras/rings R[V]^ and R[V]^ are isomorphic, we have 
the equality of the minimal degrees May = Mny- 

The third operation requires two groups Gi < GL{Vi) and G 2 < GL{V2) and 
replaces them by their direct product Gi x G 2 embedded in a natural way into 
GL{Vi 0 V 2 ). In this case the isomorhism R[Vi 0 1 / 2 ]*^^— R[Vi]^^ 0 i?[V 2 ]'^^ 
implies Ma-^xG 2 yi®V 2 = Ma^y^}, thus the minimal degree will not 

increase. 

Finally, the fourth operation replaces G by the wreath product L = GlH, where 
H is a, subgroup of the symmetric group Sm- The group L can be identified with 
the set of all m + 1-tuples [gi, ..., where gi, ..., g^ G G and a € H. The 

above element of L acts on I/®™ by the rule 


(ffl T ■ ■ ■ 1 9rai <t) (vi , • ■ . , Vm) — (5l^cr(l) j • ■ • i 9m'Va(m)) ■ 


The subgroup consisting of all elements with ct = 1 is normal and it is isomorphic to 
the direct product G™ and L is isomorphic to the semi-direct product HkG^. Then 


^ = {{R[Vf)^"^)^ and, for any invariants /i,...,/™ 


from RiyY^, the element 



0... 0 I 


i-th place 


is L-invariant. Therefore < May. 

Summing up, all four operations as presented in m do not increase the minimal 
degrees of invariants of given representations of the initial groups (possibly with the 
exception of the first operation with non-injective map tt). Therefore, regardless 
of how complicated the resulting group G and its representation is, it is no more 
secured against the linear algebra attack described in subsection 14.11 and great 
care needs to be taken that the initial minimal degrees of the starting groups are 
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large enough, say of the order n. On the other hand, if the minimal degrees of 
the starting group is sufficiently large, then from the point of view of such linear 
algebra attack it is not necessary to construct a more (structurally) complicated 
group or representation. 

5.9. Invariants of supergroups. Another possible modihcation of the cryptosys¬ 
tem is obtained when the group G and its invariants are replaced by a supergroup 
and its superinvariants. A significant difference that is exhibited in this case is that 
invariants of supergroups do not have a basis consting of monomials. Thus the 
structure of the invariants of supergroups is more complicated. 

We will not go further into rather complicated details about supergroups and 
their invariants but would like to refer an interested reader to the paper |15j where 
we have obtained results in this direction and stated potential application in cryp¬ 
tosystems based on relative invariants and absolute invariants of supergroups. 

6. Cryptosystem over finite rings R 

In order to make the cryptosystem build in Section [5] more effective and easier 
to implement we will make modihcation that will work over hnite rings R instead 
of over helds F. 

The motivating example is the residue class ring R of the ring of algebraic helds 
modulo an integer m > 1. However we can consider cryptosystem over arbitrary 
hnite commutative ring R. 

6.1. Structure theory for fiuite couiuiutative riugs R. Recall the following 
structure theorems for hnite commutative, hnite local commutative rings and their 
groups of units from [18] . 

Theorem 6.1. (Theorem VI.2 of [18j ) Let R he a finite commutative ring. Then 
R is isomorphic to a direct sum of local rings. 

Theorem 6.2. (Theorems XVII.1 and XVIII.2 of [18j } Let R be a finite local 
commutative ring of characteristic p" with maximal ideal m and residue field k. Let 
[k : Zp] = r and ui,...,ut be a minimal R-generating set of m. Then the largest 
Galois extension T (called the coefficient ring of R) of Ip in R is isomorphic to the 
Galois ring GR(jF^r), and R is a ring homomorphic image of the polynomial ring 
T[Xi,...,W]. 

The group of units R^ of R is isomorphic to (1 + m) x . The Abelian p-group 
1 + m is called the one group of R. 

Gilmer has characterized when is cyclic. 

Theorem 6.3. (Theorem XVIII.9 of [18j ) Let R be a finite local commutative ring. 
Then R^ is cyclic if and only if 1 + ra is cyclic. In this case R is isomorphic to 
one of the following 

• GF{p*) (if m = 0) 

• Z/p”Z (if p > S and n > 1) 

• Z/4Z 

. (Z/pZ)[X]/(X2) 

. (Z/2Z)[X]/(X3) 

• (z/4Z)[a:]/(2a:,a:2 - 2) 
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Hence in most cases is not cyclic. 

We will work with finite rings R obtained as residue rings of the ring of algebraic 
integers Z of number helds F modulo an ideal a of and specialize further to the 
case when a = (m) and m G Z. 

The structure of units U(R) of the residue classes R ot Z modulo a power of a 
prime ideal p is rather complicated and described in Theorem 2 and 3 of [19) . For 
simplicity we quote here only Theorem 1 of m describing thep-rank of U (Z/p^^^). 

Let p be a prime ideal of Z dividing a prime number p of Q, e and / be the 
ramification index and the degree of p over Q. Denote by ei = [^;^]. The following 
statement is due to Hasse and Takenouchi. 

Theorem 6.4. (Theorem 1 o/ [19]^ The p-rank Rn of U{Zis given by 

• (^-[f])/ */0<7V<e + ei 

• ef if N > e -\- ei and the primitive p-th root of unity does not belong to the 
p-adic completion Fp 

• ef 1 if N > e ei and the primitive p-th root of unity belongs to the 
p-adic completion Fp. 

An interesting connection of the structure U{Zla) to multiplicative semigroups 
and Fermat-Euler theorem in algebraic number held is given in m- 

6.2. Modification of the cryptosystem to finite rings. Assume that F is a 
number field, that is a finite extension of the field Q of rational numbers. Let Z be 
a ring of algebraic integers of F. 

Let a = p^”^ ... p^'', where pi for i = 1,... r are distinct prime ideals of Z and 
ki are the corresponding multiplicities. We say that elements a, (3 G Z are called 
congruent modulo a, and write a = (3 (mod a) if a — /3 is divisible by a. The 
equivalence classes of this congruence form a residue class ring R = Zja. Then R 
is a finite commutative ring that is isomorphic to a direct sum of local rings Zjp\'. 
If p has the degree of inertia /, then the norm Norm{p) = p-^ for a rational prime p 
and the cardinality of the local ring Z/p’^ is p-^^. In particular, the residue class ring 
Z/p is a finite field of cardinality q = p^. This way we have a concrete realization of 
the structure theorem for finite commutative rings for residue class rings R = Zja. 

There is a theory of divisors on R induced from the theory of divisors on Z. The 
finite ring R has divisors of zero and the ideals of R are in one-to-one correspondence 
to ideals of Z dividing a. Thus R has r distinct prime ideals ^ for i = 1,..., r and 
the product p^^ ... vanishes. 

6.2.1. Choice of the ring R and its ideal a. Let us make the following observation. 

Proposition 6.5. Let Z he the ring of algebraic integers of a number field F, and 
a he an ideal of Z. Then the residue class ring R = Zla. is a principal ideal ring 
that is isomorphic to a finite product of local rings. 

Proof. Let a = p^^ .. .pji'', where pi for i = 1,.. .r are distinct prime ideals of Z 
and ki are the corresponding multiplicities. 

For each z = 1,..., r denote by Zi the localization of Z with respect to the prime 
ideal pi and by i?i = Zi/p’^' the local ring with the maximal ideal piRi. It follows 
from the Chinese remainder theorem that the map (f : R ^ 01=1 that sends x 
(mod a) to {x (mod pj^0)i=i is an isomorphism of rings. Moreover, the groups of 
units U{R) of R and U{YYi=i ^i) isomorphic under this map. 
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Ideals b of i? are of the form where 0 < k < ki for each i = 1,... ,r. 

If we choose elements tt^ such that tt G pi \ p^ (tt^ uniform element with respect 
to the valuation of Z corresponding to pi) and TTi = 1 (mod pj) for j i (this is 
possible by the Chinese remainder theorem), then b = (01=1 

Let us note that the groups of units U{R) of R and U{JYi=i A) = IILi U[Ri) 
are also isomorphic under the map 4>, and each group U{Ri) is cyclic and isomorphic 
to the group of units of the finite held Zjpi. 

It might appear that due to the above Proposition, when we pass from the ring 
Z that is not a unique factorization domain (when its class number h > 1) to the 
residue class ring R = Zja, the bad properties of factorization in Z do not carry 
over to R since i? is a principal ideal ring. 

However, the complexity of the divisor theory of i?, together with a suitable 
choice of a, indeed influences the complexity of the structure of R. If the number r 
of primary factors p^" and/or the exponents ki are high and the factorization of a 
is not available, it will be difficult to derive the structure of R effectively from the 
ring Z and its ideal a. 

Determining the abstract structure of an arbitrary hnite ring R seems to be even 
more complicated problem. 

Based on this disussion, we should choose the residue ring R and its ideal a in 
such a way that the number r of primary components of a is high and the prime 
factorization p^^^ ... pr^'’ of a in Z is complicated. 

We will consider only the case when R = Zl{m) for the ring of algebraic inte¬ 
gers of a number field F and integer m > 1. The choice of F and m cannot be 
independent since even if Z has a complicated theory of divisors, choosing wrong 
m can create R that is rather simple. 

We will now modify previously defined cryptosystem based on invariants of diag- 
onalizable matrices to the case when R is the residue class ring R = Zl{m) where 
Z is the ring of algebraic integers of a number field F and m is an integer. 

However, we start with F with complicated divisor theory and then look for 
appropriate m so that the structure of the residue ring Z/ (m) contains the compli¬ 
cation of the factorization in Z and also problem of the factorization of the modulus 
m. 

Let m = .. .p/'’ and each pi decomposes as pi = p^Y ■ • • in Z. Let us 

consider the following cases for m. 

1) m is square-free. In particular, one appealing choice is when m is a product 
of two large primes pi and p 2 ■ In this case the ring R will be a direct sum of finite 
fields corresponding to residue class rings of Z by unramified prime divisors of m. 
We should choose the prime factors of m to be unramified in F because otherwise 
it would be easier to factor m by considering the greatest common divisor of m and 
the discriminant of Z. Special case to consider is when either pi, p 2 or both are 
totally unramihed. 

If we want to involve local rings that are not finite fields in the decomposition 
of R, we have additional choices for m. 

2) We can choose m that is not square-free. For example, we can choose m = 
{piP 2 )^, where both pi and p 2 are large primes. In this case we have the factorization 
problem for pip 2 but since m is not square-free, it is much easier to find prime 
factorization of m when compared to the square-free case. 
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If we consider a more general case and replace the integer m by an ideal a 
(principal or not) of Z such that a = .. •p^'', then the structure of U{Zla) is 

even more complicated because the group of units of the local rings Zi/(p v) might 
have high p-rank if the exponents ki are high. 


6.2.2. Choice of the group G. Assume that we have already chosen Z, m and the 
corresponding complicated residue ring R = Zf{m). Let us modify the cryptosys¬ 
tem introduced in 15.11 in such a way that instead of working inside the ring Z we 
will work inside the residue ring R = Zj(m). 

We will assume that the entries in the matrices representing generators pi of 
G are units of R. Assume that the group of units U of the ring R has a basis 
of respective orders oi,..., Or, generators gi of G are given as 

o\ 

0 I- j 

for i = 1,... t, where 


given by Ml,... 
frn 0 
0 r^2 


= 


\ 0 0 


V 


exponents hjk, and the set S consists of vectors vt = 


I OiA 
012 

yOm J 


.. for appropriate 


for i = 1,..., s. The 


general element of G is written as p = g^ ... g^ for some integers yi , 
A monomial / = separates Vi and Vj if and only if 


,yt- 


aiia,2 


tJjO ^TL 


Since every invariant of C? is a sum of monomial invariants, to obtain a complete 
description of all invariants of G we only need to find monomial invariants. A 
monomial / = is an invariant of G if ^ 

all integers yi,... ,yt. This implies rffrff ... rffl = 1 for each z = 1,..., t. 

For fixed z, the condition = 1 gives zzp^^^ ^ 

and that is equivalent to the system of congruences 


dihik + d 2 h 2 k + • ■ ■ + dnknk = 0 (mod Ok) 


for each k = 1,..., r. 

Since z runs from 1 to t, the condition that / is an invariant of G is equivalent to t 
such systems. In total we obtain a system of rt congruences in variables di,..., dn- 
If we assume that numbers lijk are determined, then the last system can be solved 
using integer programming since it is equivalent to the system of rt equations 


dlhlk + d2li2k dnhnk + Okdik — 0 


for each z = 1,..., t and fc = 1,...r in integer variables di,... ,dn and rt variables 
dik. 

When we worked over the finite field F = GF{q) we were able to determine the 
corresponding coefficients Uj using discrete logarithms. In the case of finite rings 
R we do not have such a tool at our disposal. If we want to determine coefficients 
lijk, first we need to determine the structrure of the ring R and its group of units. 
Papers [53] and [1] provide algorithms that perform this task with the complexity 
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Even when the basis ui,... ,Ur of C/ is known it is not clear how to determine 
the coefficients hjk. This issue can make the use of the cryptosystems over finite 
rings more appealing. 

Important note is that all the difficulties we have just described are needed for 
breaking of the cryptosystem. However for the design of the cryptosystem we do 
need to know neither a basis ui,... ,Ur of U nor the exponents lijk- All that is 
required is to check that the diagonal entries are units in the residue ring R. Since 
our ring R is the residue class of Z modulo m, this condition can be verified by 
computing the norm of these entries. If all the norm of the entry is coprime to m, 
then its image in R belongs to U. 

6.2.3. Conclusion. The breaking of the modified cryptosystem designed over R 
seems to require techniques going beyond discrete logarithm problem. More work 
that is required to specify the parameters of the cryptosystem that provide its 
sought-after security is beyond the scope of this paper. We hope that we have 
convinced the reader that this is a worthwhile endeavour to undertake. 

In the papers [10] and m the cryptosystem based on invariants of groups over 
a field F were considered. The above modification of our cryptosystem works over 
finite rings instead of fileds. The reason why we need the ring structure is because 
we want to use the matrix multiplication and conjugation by a matrix P to hide 
the group G as suggested by Grigoriev. If it is determined that the conjuagation 
by P does not increase the security of the cryptosystem, then we can consider a 
more general setup and instead of working over finite rings we could work over finite 
multiplicative groups. That is another direction for future investigation. 
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